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Abstract: 


Multi-modal  data  fusion  techniques  are  commonly  used  to  enhance  decision-making  processes. 
In  previous  research,  a  comprehensive  structural  analysis  process  was  developed  for  quantizing 
and  evaluating  characteristics  of  defects  in  aircraft  lap-joint  mimics  using  eddy  current 
nondestructive  evaluation  (NDE)  data  collected  for  structural  health  monitoring.  In  this  research, 
a  comprehensive  multi-modal  structural  analysis  process  is  presented  that  includes  intra-  and 
inter-modal  NDE  data  fusion.  The  process  includes  defect  detection,  defect  characterization  and 
Finite  Element  Modeling-based  simulated  fatigue  testing  for  structural  analysis.  The  multi-modal 
structural  analysis  process  is  then  experimentally  evaluated  using  five  lap-joint  mimic  test  panels 
with  corroded  patches  at  different  layers  of  the  lap-joints  and  one  painted  pristine  panel  used  as  a 
reference.  The  test  panels  are  subjected  to  two  rounds  of  mechanical  loading,  preceded  by  eddy 
current,  millimeter  wave,  and  ultrasonic  NDE  data  obtained  prior  to  each  round.  Different 
modality  combinations  are  used  to  model  the  test  panels  at  the  different  rounds  of  measurements. 
Experiments  are  performed  to  compare  the  simulated  fatigue  loading  based  on  models 
determined  from  different  modality  combinations  and  the  mechanical  loading  results  to  find 
susceptible-to-failure  areas  in  the  test  panels.  The  results  show  that  the  comprehensive  structural 
analysis  process  found  the  correct  locations  of  failure  areas  at  rates  as  high  as  98.8%,  using  eddy 
current  and  ultrasonic  modalities. 

Index  terms:  Multi-Modal  Data  Fusion,  Nondestructive  Evaluation,  Structural  Health  Monitoring. 
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I.  Introduction: 


Corrosion-induced  aircraft  structural  degradation  is  the  subject  of  constant  attention  and 
concern  for  those  involved  in  maintaining  operational  integrity  and  safety  of  aging  aircraft  fleets 
(i.e.,  commercial,  cargo  and  military).  Hidden  corrosion  under  paint  or  in  between  various  lap- 
joint  layers  tends  to  facilitate  early  fatigue  crack  initiation  and  enhanced  crack  growth  when  the 
aircraft  experiences  in-service  loadings  [1,2]  .To  this  end,  the  U.S.  Air  Force  has  proposed  a 
“predict  and  manage”  scheme  for  corrosion  management  as  a  deviation  from  the  current  “find-it 
and  fix-it”  approach  [3]. 

To  characterize  and  evaluate  corrosion  in  aging  aircraft  structures  many  nondestructive 
evaluation  (NDE)  methods  have  been  explored  in  the  past  and  are  currently  being  implemented 
[4-11],  In  general,  NDE  methods  may  provide  qualitative  information  (e.g.,  corroded  or  not) 
and/or  quantitative  evaluation  (e.g.,  corrosion  thickness/severity,  spatial  extent,  etc.)  about  a 
structure.  Also,  each  NDE  method  used  for  corrosion  detection  has  its  own  limitations  and 
advantageous  features  as  outlined  in  detail  in  [8].  Some  of  the  more  prominent  methods  for 
aircraft  structural  integrity  analysis,  and  more  specifically  for  corrosion  evaluation,  include  eddy 
current  (EC),  ultrasonic  (UT),  optical,  thermography,  radiography  and  millimeter  wave  (MW) 
methods  [8], 

EC  inspection  systems  are  small,  portable  and  inexpensive,  and  are  capable  of  providing 
effective  metal  loss  information.  Both  pulsed  and  conventional  EC  NDE  methods  have  been  used 
for  corrosion  detection  in  aging  aircrafts  [5-7].  Conventional  EC  methods  can  be  used  to  provide 
structural  metal  loss/thinning  information,  while  pulsed  EC  methods  provide  material  loss  depth 
information  as  well  [10], 
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Ultrasonic  methods  are  effective  inspection  tools  for  a  number  of  NDE  applications  and 
employ  high  frequency  sound  waves  (with  relatively  short  wavelengths  in  the  material). 
Ultrasonic  signal  amplitude,  frequency  or  time  of  flight  (ToF)  may  be  used  to  detect  and  evaluate 
a  number  of  defect  characteristics.  UT  can  be  limited  in  its  capability  to  distinguish  features  in 
subsequent  structural  layers  underneath  the  surface  layer.  This  is  due  to  the  lack  of  an  effective 
UT  coupling  media  between  layers.  In  contrast,  EC  is  not  limited  by  these  constraints,  has  a 
relatively  high  signal  to  noise  ratio,  and  is  able  to  provide  depth  information  as  well  [4,  10]. 

Millimeter  wave  NDE  methods  offer  several  important  advantages  for  detection  and 
evaluation  of  corrosion  under  paint  and  thick  composite  laminate  coatings  [9,  12,  13].  Millimeter 
wave  signals  can  penetrate  inside  of  low-loss  dielectric  materials  and  interact  with  their  inner 
structure.  They  are  also  sensitive  to  changes  associated  with  dielectric  properties  and  boundary 
interfaces,  which  make  them  very  attractive  for  detecting  the  presence  of  undesired  layers  such 
as  corrosion  under  paint.  Near-field  MW  NDE  techniques,  employing  open-ended  rectangular 
waveguide  probes,  have  been  successfully  used  to  detect  the  presence  of  corrosion  and  corrosion 
precursor  pitting  under  paint  and  primer  in  both  steel  and  aluminum  substrates  [9,  12-17], 
However,  MW  signals  do  not  penetrate  inside  of  electrically  conducting  materials  such  as  metals 
and  can  therefore  only  provide  information  about  surface  flaws  such  as  pitting  and  corrosion 
under  paint  or  dielectric  coatings.  Therefore,  as  it  relates  to  the  objectives  of  this  investigation 
and  when  considering  a  lap  joint  made  of  several  aluminum  panels,  other  methods  must  be 
employed  to  detect  corrosion  in  between  the  various  layers  of  the  lap  joints. 

Results  of  previous  research  have  clearly  demonstrated  that  an  effective  way  to  overcome 
the  limitations  of  each  NDE  method  is  to  integrate  several  of  these  NDE  data  using  multi-modal 
data  fusion  techniques.  In  this  way  the  complementary  information  across  the  various  NDE 
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modalities  can  be  effectively  combined  to  provide  significantly  more  information  than  any  one 
method  can,  resulting  in  substantial  increase  in  measurement  reliability.  Using  complementary 
information,  data  fusion  techniques  have  the  potential  to  provide  a  much  better  and  more 
comprehensive  understanding  of  the  physical  process,  which  in  this  case  is  corrosion,  as 
compared  to  using  one  sensor  data  or  using  multiple  sensors  in  an  independent  manner  [4,  8,  10, 
18].  Data  fusion  provides  a  mathematical  framework  to  enhance  the  data  analysis  process  based 
on  data  collected  from  a  single  modality  and  processed  using  multiple  methods  (intra-modality 
fusion)  and/or  multiple  modalities  processed  using  one  or  more  techniques  (inter-modality 
fusion).  The  context  for  data  fusion  in  this  investigation  is  to  “measure”  (i.e.,  assess  and  predict) 
corrosion-induced  crack  locations  in  aircraft  lap  joints.  Pixel  level  multi-modal  NDE  data  fusion 
techniques  have  been  applied  using  probabilistic  and  wavelet  methods  [19,  20],  and  simple 
evidence  theory  as  in  [21].  Some  NDE  data  fusion  works  involving  fusing  EC  and  UT 
measurement  data  have  been  outlined  in  [19,  22-25], 

The  first  step  in  this  investigation  involved  a  comprehensive  structural  analysis  process 
for  identifying  vulnerable  regions  in  aircraft  lap-joint  mimics  using  eddy  current  NDE  data  only 
[11].  That  study  introduced  a  framework  for  systematic  analysis  of  the  test  panels,  from  data 
collection  to  using  this  data  for  failure  assessment  of  the  test  panels.  The  structural  analysis 
process  included  quantizing  and  measuring  characteristics  of  aluminum  aircraft  lap-joint  mimics 
based  on  EC  NDE  data  acquisition,  defect  detection  and  characterization,  three-dimensional  (3- 
D)  structural  model  generation,  finite-element  modeling-based  (FEM)  simulation  of  fatigue 
damage  for  structural  analysis  [11].  This  research  extends  the  structural  analysis  process  to 
multi-modal  NDE  data  collection  and  intra-  and  inter-modal  data  fusion  using  image  and  signal 
processing  and  computational  intelligence  techniques  for  lap-joint  mimic  aircraft  panel  analysis. 
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Fusion  of  different  NDE  modalities  (eddy  current,  millimeter-wave  and  ultrasonics)  with 
complementary  informational  properties  is  performed  to  provide  a  comprehensive  snapshot  of 
the  corrosion  environment  in  these  panels.  This  paper  presents  the  comprehensive  structural 
analysis  process  investigated  based  on  models  determined  for  different  modality  combinations 
for  simulated  fatigue  testing  for  comparison  with  actual  fatigue  testing  for  five  lap-joint  mimic 
aircraft  test  panels. 

II.  Methodology 

A.  Overview  of  Aircraft  Lap-Joint  Mimic  Panel  Description  and  Preparation 

A  set  of  five  aluminum  lap-joint  mimic  test  panels,  four  of  which  were  corroded  and  one  pristine, 
were  used  for  the  NDE  data  collection  and  data  fusion-based  measurement  process.  These  panels 
were  constructed  using  ten  A1  2024-T3  aluminum  plates  with  dimensions  of  254  mm  (10")  by 
76.2  mm  (3")  with  a  nominal  thickness  of  1.6  mm  (0.063").  Two  aluminum  plates  were  stacked 
and  riveted  to  create  a  lap-joint  mimic  test  panel.  Thus,  ten  aluminum  plates  produced  five  test 
panels.  Prior  to  stacking  and  riveting,  the  aluminum  plates  of  the  corroded  test  panels  were 
subjected  to  accelerated  corrosion  in  a  salt  fog  chamber  following  the  ASTM  B1 17  test  standard. 
The  ASTM  B1 17  standard,  designed  in  1939,  is  a  widely  used  technique  for  evaluating  the 
relative  corrosion  of  various  metals  and/or  coatings.  The  salt  spray  chambers  designed  according 
to  this  standard  are  automated  to  maintain  a  specified  environment  within  the  chamber  [26]. 
Details  of  the  panel  description  and  preparation  have  been  reported  in  detail  in  [1 1],  Figure  1 
shows  one  of  each  set  of  painted  pristine  and  corroded  panels,  with  the  close-up  views  showing 
the  riveted  region  of  the  panels. 
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Figure  1:  Picture  of  painted  pristine  and  corroded  panels  with  rivets. 


B.  Overview  of  NDE  Data  Collection  and  Test  Panel  Fatigue  Testing 

For  this  investigation,  three  different  NDE  modalities  were  used;  namely,  EC,  UT  and  MW 
methods.  The  test  panels  were  subjected  to  two  rounds  of  fatigue  testing,  with  NDE  data 
collected  prior  to  each  round.  The  fatigue  testing  was  performed  for  comparison  with  the 
simulated  fatigue  loading  analysis.  During  fatigue  testing,  the  test  panels  were  cyclically  loaded 
between  1.04  kN  and  10.4  kN  (load  ratio,  R  =  Pmin/Pmax  =  0.1)  at  a  frequency  of  4  Hz.  The 
calculated  stress  at  the  maximum  load  (10.4  kN)  in  the  unriveted  portions  of  the  panels  was  82.7 
MPa.  The  first  round  of  fatigue  testing  was  intended  to  initiate  small  fatigue  cracks,  wherein,  the 
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test  panels  were  cyclically  loaded  until  a  crack  was  noticed  (see  Table  1).  For  the  second  round 
of  fatigue  testing,  cracks  were  allowed  to  grow  to  twice  their  former  lengths  or  if  no  cracks 
originated  in  the  first  round,  cracks  were  initiated  and  allowed  to  grow  to  a  length  of  ~3  mm. 
The  panels  were  photographed  using  a  stereomicroscope  to  inspect  and  measure  crack  lengths 
after  each  round.  These  cracks  were  meant  to  simulate  in-service  flaws.  Details  of  the  fatigue 
testing  process  (also  referred  to  as  the  actual  mechanical  testing  process),  is  provided  in  detail  in 
[11].  Figure  2  shows  the  flowchart  of  the  NDE  data  collection  and  fatigue  testing  steps  for  each 
round. 


Panels  1-4, 
Pristine  Panel 


t 


Figure  2:  Flowchart  of  the  NDE  data  collection  and  fatigue  testing  steps  for  each  round. 
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C.  Multi-Modal  Data  Fusion  Process  Overview 

Figure  3  presents  the  multi-modal  NDE  multi-modal  comprehensive  structural  analysis  process 
used  in  this  study  to  detect  test  panel  vulnerabilities,  extending  the  previously  investigated 
process  involving  only  uni -modal  EC  NDE  data  [11],  The  various  steps  in  this  comprehensive 
process  consist  of:  1)  acquisition  of  NDE  data  for  each  modality,  2)  acquisition  of  fatigue  testing 
data,  3)  intra-modal  NDE  data  fusion  for  defect  detection,  4)  multi-modal  fusion  of  the  defect 
detection  information,  5)  defect  characterization  including  material  loss  estimation,  6)  3-D  model 
generation  incorporating  detected  defects  using  the  multi-modal  data  fusion  approach,  and  7) 
simulated  fatigue  testing  for  structural  analysis  and  evaluation  of  the  test  panels  with  actual 
fatigue  testing. 
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Figure  3:  Overview  of  the  multi-modal  structural  analysis  process. 
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Let  Xm>, X1"1  denote  the  registered  images  obtained  for  each  modality  for  a  test 

panel  (see  Figures  3  and  4),  where,  m1,m2,...,mn denote  the  n  different  modalities.  Registering 
the  different  modality  images  for  a  test  panel  was  performed  by  manually  selecting  the  rivet 
positions,  and  an  affine  transformation  was  applied  to  generate  images  of  standard  size  for  the 
comprehensive  structural  analysis  process  (see  Figure  3). 

The  next  step  in  the  process  is  corrosion  detection.  In  the  previous  study,  an  Rx 
statistical  method  (Rx  Detector)  and  a  data-driven  fuzzy  logic-based  (Fuzzy  Detector)  data 
fusion  method  was  developed  for  corrosion  detection  [11,  27-29],  In  [1 1],  the  data  fusion  method 
involved  obtaining  the  decision-level  fusion  image  (representing  detected  corrosion)  from  EC 
images  only.  In  this  study,  a  two-step  data  fusion  approach  was  developed  which  can  combine 
NDE  data  obtained  from  multiple  modalities  to  create  a  final  decision-level  fusion  image  (see 
Figure  4).  First,  the  multi-modal  fusion  between  the  results  of  the  individual  anomaly  detectors 
(Rx  and  Fuzzy  Detectors)  was  determined  to  create  the  fused  detector  level  image.  Second,  the 
decision-level  multi-modal  fusion  of  the  Rx  and  Fuzzy  Detector  images  to  produce  the  multi¬ 
modal  decision-level  fusion  image. 
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\m]  ,m2  } 


Figure  4:  Various  modality  combinations  used  in  the  multi-modal  corrosion  detection  algorithm 
The  Rx  and  Fuzzy  detector  images,  as  applied  to  the  original  modal  images,  are  denoted 

as  X2,X2,-..,  X';;  and  XF'uzzy ,  X%zzy XF”zzy ,  respectively.  The  Rx  and  Fuzzy  detector 

algorithms  are  given  in  detail  in  [1 1],  The  next  step  involved  the  fusion  of  the  detector  images 
using  a  maximum- likelihood  based  method  [30],  which  is  given  in  Equations  (1)  and  (2),  where 

Y fix'1  n‘2  '  and  YF^Z"'2  '"'" '  denote  the  fused  Rx  detector  and  Fuzzy  detector  images  for  the 


modality  combination  respectively,  while  cr^,cr^,....,cr^  and  <j"puzzy  , oFizzy 

cr fuzzy  denote  the  corresponding  Rx  and  Fuzzy  detector  standard  deviations  from  application 
to  the  original  images,  respectively. 
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represent  the  standard  deviation  for  the  fused  Rx  and  fused 


Fuzzy  Detector  images,  respectively.  Hence,  the  multi-modal  decision-level  fusion  image  is 


represented  as  D 


,  and  obtained  as  shown  in  Equation  (3). 


D 
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{m^,. .,»»„}  2  +  ,  {mvm2,..,mn}  2 
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(3) 


Once  the  decision-level  fusion  image  is  obtained  for  corrosion  detection,  the  next  step  in  the 
comprehensive  structural  analysis  process  is  to  compute  the  material  loss  estimation  image, 

denoted  as  F{m"mi (see  Figure  3).  The  material  loss  estimation  image,  as  presented  in  detail 
in  [23]  is  obtained  using  interpolation  of  cluster  distance  from  known  material  loss  signatures  of 
0%,  10%,  15%  and  20%  determined  from  a  reference  panel  [1 1],  A  sample  schematic  of  a 
reference  panel  is  shown  in  Figure  5. 

•  Q 

20%  corrosion  15%  corrosion  10%  corrosion  0%  corrosion 

(background) 

Figure  5:  Sample  schematic  of  a  reference  panel  is  used  for  the  material  loss  estimation. 
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Using  the  material  loss  estimation  technique  explained  above,  a  3-D  geometrical  model 
representation  for  the  riveted  test  panels  is  generated.  A  localized  non-overlapping  windowing 
method  was  used  for  model  generation,  presented  in  detail  in  [1 1], 

III.  NDE  Data  Acquisition 

The  process  and  sources  of  NDE  data  collected  for  the  test  panels  and  the  actual  fatigue  testing 
process  used  for  analyzing  the  test  panels  are  presented  in  this  section. 

B.  Panel  Data  Collection 
1)  Eddy  Current  Data 

To  reveal  corrosion/defects  in  the  test  panels,  EC  measurements  were  performed  on  each 
of  the  panels  for  two  rounds  of  data  collection.  This  data  collection  was  performed  at  the  Air 
Force  Research  Laboratory,  OH,  where  images  of  the  test  panels  were  obtained  from  the  painted 
side  only  since  this  is  the  accessible  side  on  an  aircraft.  The  EC  data  collection  was  performed 
using  the  Boeing  MAUS  (Mobile  Automated  Scanner)  raster  scanning  system,  with  phase  lag 
adjustment  to  discern  defect/corrosion  depth.  The  inspected/imaged  area  covered  the  corroded 
regions  containing  the  rivets,  as  shown  in  Figure  1.  The  details  of  the  EC  data  collection  are 
presented  in  detail  in  [1 1],  Figure  6  shows  an  example  of  such  an  EC  data  collection  image. 


O  O 

o  o 

U  o 


Figure  6:  EC  image  of  a  corroded  test  panel. 
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2)  Ultrasonics  Data 

Ultrasonics  NDE  data  was  also  collected  at  the  Air  Force  Research  Laboratory,  OH,  in  a 
similar  fashion  as  those  for  EC.  Measurements  were  conducted  at  a  frequency  of  15  MHz  and 
images  were  produced  using  a  step  size  of  0.015"  [29].  As  presented  in  Section  II. A,  each  test 
panel  comprises  two  aluminum  plates,  the  top  plate  and  the  bottom  plate.  In  Figure  1,  the  picture 
of  the  top  and  bottom  sides  of  the  test  panel  are  shown.  During  the  UT  NDE  data  collection 
process,  a  reference  panel  image  was  obtained,  representing  0%,  10%,  15%,  and  20%  material 
loss,  as  shown  in  Figure  7  (a).  In  Figure  7(a),  physically,  for  example,  the  pixels  contained  in  the 
15%  corrosion  section  represent  15%  deviation  from  the  pixels  contained  in  the  0%  corrosion 
region.  These  reference  panel  images  were  created  to  have  baselines,  representative  of  material 
loss  in  the  test  panels.  They  were  primarily  used  for  calibration  purposes  for  corrosion  detection 
and  material  loss  estimation  algorithm  development.  Figure  7(b)  shows  a  UT  image  of  the  lap 
joint  riveted  region,  as  that  used  to  produce  the  EC  data  shown  in  Figure  6. 


Figure  7:  (a)  Reference  panel  used  for  calibration  purposes  with  areas  of  various  levels  of 
removed  metal;  (b)  UT  image  of  corroded  test  panel  taken  at  15  MHz. 

3)  Millimeter  Wave  Data 

The  MW  images  were  produced  at  the  Applied  Microwave  Nondestructive  Testing 
Laboratory,  Missouri  S&T.  A  MW  reflectometer,  operating  at  67  GHz  (V-band)  consisting  of  a 
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continuous-wave  (CW)  MW  source  (Gunn  oscillator),  a  magic-tee  (i.e.,  power  splitter  and 
combiner),  and  a  diode  detector  was  used,  as  shown  in  Figure  8.  Through  extensive 
measurements  and  considering  trade-offs  such  as  detection  capability  vs.  spatial  resolution,  V- 
band  (50-75  GHz)  was  shown  to  be  a  suitable  frequency  range  for  this  investigation.  As  shown 
in  Figure  8,  a  portion  of  the  incident  signal  irradiates  the  sample  through  an  open-ended 
rectangular  waveguide  probe  with  dimensions  of  3.75  mm  by  1.87  mm.  The  reflected  signal  is 
routed  to  the  detector  input.  In  addition,  a  portion  of  the  incident  signal  is  also  routed  to  the 
detector  input  via  a  short  circuit  (e.g.,  reference  signal)  at  the  remaining  port  of  the  magic-tee.  In 
this  way  the  reflected  and  reference  signals  combine  at  the  input  to  the  detector.  In  this  way,  the 
detector  behaves  like  a  mixer  and  the  detector’s  DC  output  voltage  will  be  proportional  to  the 
magnitude  and  phase  of  the  reflected  signals.  Given  the  small  amount  of  expected  corrosion,  the 
phase  of  the  reflected  signal  is  mostly  sensitive  to  the  presence  of  corrosion.  Subsequently,  the 
lap  joint  panels  were  scanned  by  this  reflectometer  at  certain  standoff  distance,  producing  MW 
images  of  the  panels.  These  images  in  general  are  quite  suitable  for  quantitative  analysis,  such  as 
estimation  of  corrosion  thickness,  and  data  fusion  with  other  modalities.  Images  were  produced 
at  67  GHz  for  corroded  and  pristine  panels  corresponding  to  scanned  areas  of  80  mm  by  55  mm, 
at  step  size  of  2  mm  by  1  mm,  and  at  a  standoff  distance  of  0.5  mm.  These  images  were 
subsequently  analyzed  to  determine  the  presence  of  corrosion  and  to  evaluate  corrosion 
thickness.  It  was  found  that  the  top  12  to  15  mm  of  each  image  consisted  of  non-corroded  area, 
so  that  this  information  may  be  used  as  reference.  Since  the  measurements  were  affected  by  the 
presence  of  sharp  edges  of  the  samples,  using  two  polarizations  (see  Figure  9)  yielded  a  better 
estimate  of  material  loss.  The  technique  for  measuring  the  material  loss  (due  to  corrosion)  is 
outlined  in  [15]  in  which  a  first  order  approximate  model  for  the  probe,  not  taking  into  account 
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multiple  reflections,  was  developed.  In  this  investigation,  the  parameters  of  the  probe  were 
directly  measured  using  a  performance  network  analyzer.  Furthermore,  the  actual  detector’s 
input-output  curve  (from  the  datasheet)  was  used  instead  of  assuming  a  square-law  region  of 
operation  for  more  accurate  model  compared  to  [15].  As  a  result  a  relatively  accurate  model  was 
obtained  that  converts  a  reflection  coefficient  at  the  aperture  (5)  (see  Figure  8)  of  the  probe  to  a 
voltage  output  at  the  detector.  The  theoretical  reflection  coefficient  at  the  aperture  of  the  probe 
was  calculated  using  the  model  for  open-ended  waveguide  radiating  a  multilayer  structure  (e.g., 
paint-corrosion-conductor)  [16].  The  dielectric  constants  of  similar  paint  and  corrosion  layers 
were  previously  measured  in  [31].  Ultimately  a  curve  was  constructed  relating  the  amount  of 
corrosion/metal  loss  to  the  voltage  output  of  the  probe. 

Reference 


Figure  8:  Schematic  of  the  V-band  millimeter  wave  reflectometer. 
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(a) 


(b) 

Figure  9:  Picture  of  a  sample  with  marked  scanned  area  and  orientation  of  probe  aperture  at  (a) 

parallel  and  (b)  perpendicular  polarization. 

The  pixel  value  of  every  data  point  in  these  images  is  equal  to  the  output  voltage  of  the 
reflectometer  at  that  particular  location.  For  the  MW  images,  each  pixel  location  represented  the 
thinning  that  occurred  due  to  corrosion  at  that  particular  location.  Therefore,  the  reference  values 
for  calibration  purposes  (analogous  to  reference  panels  explained  earlier)  to  be  used  for  material 
loss  estimation  was  computed  from  the  MW  images  directly  using  a  scaling  approach  to  find 
pixel  values  that  would  represent  the  10%,  15%  and  20%  material  loss  areas  in  the  MW  images. 

B.  Multi-Modal  Data  Fusion  Algorithms 

In  this  study,  we  investigated  the  multi-modal  data  fusion  algorithm  (presented  in  Section 
II.C)  as  applied  to  three  separate  multi-modal  fusion  cases:  {EC,UT},  {EC,  MW}  and, 

{EC,UT,MW}.  If  XEC ,  XUT  and  XMW  denote  the  images  that  are  obtained  using  the  three 

different  modalities,  then  XEE ,  XEE  and  XR'V  Let  XEE ,  XEE  and  XR'V  represent  the  Rx 

Detector  images  of  the  EC,  UT  and  MW  modalities,  respectively.  Similarly,  XEEzzy ,  XuFTuzzy  and 


- Scan  area 

I  I  Probe  aperture 

Polarization  direction 
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Xp^y  represent  the  corresponding  Fuzzy  Detector  images.  Then,  the  fused  Rx  Detector  images 


are  represented  as  Y^CUT< ,  y^C  MW]  and  y^cut,mw}  (Equations  (4)-(6))  for  the  {EC,UT}  , 
{EC, MW} and,  {EC,UT,MW}  cases. 


y{EC,UT} 

JRx 


/  £C\2  v EC  .  /  6T\ 2  xrUT 

)  XRx  +(^x)  XRx 
/  £C\  2  ,(-UT\  2 

V7**)  +(Crto) 


(4) 


y{EC,MW } 
*Rx 


(  —EC \2  VEC  .  s^MW\ 2  VMW 
(°Rx}  XRx  +  (°Rx  ) 

(  —EC  \2  ,/-.MW\2 
\GRx)  +(Cri?x  ) 


(5) 


y  {EC  ,UT  ,MW  } 
1Rx 


s  EC  \2  xr EC  .  /  Z7T\ 2  .  (  —MW  \2  VMW 

[Em  1  Em  22Em1  Em  IeEm  2  ^ 

/_£C\ 2  .  f—UT \2 

\°1 tv)  +(CrRv)  +  (<7K*  ) 


(6) 


Similarly,  the  fused  Fuzzy  Detector  images  are  represented  as  Y^zzyUT} ,  Y^zzyMW}  and 
(Equations  (7)-(9)). 


y{EC,UT } 
1  Fuzzy 


(  EC  \2  /  i/r 

v°  f  ^  Fuzzy  ^  Fuzzy 

f  —EC  \2  ,  (  —UT 
y  ® Fuzzy  '  v  ^ Fuzzy 


x2  YC/J 

/  ^  Fuzzy 

f 
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Fuzzy 
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y{EC,UT,MW} 
*  Fuzzy 


Fuzzy  > 


fXZ,+(r 


MW  y  MW 


Fuzzy  > 


Fuzzy 


(  EC  \2  ,(—UT  \2,(—MW  \ 

y*7 Fuzzy  )  ^ Fuzzy  )  +  '  &  Fuzzy  ) 


jW  \2 


(9) 


The  decision-level  fusion  images  were  obtained  utilizing  the  detector  level  fusion  images  using 
the  same  maximum  likelihood  scheme  defined  earlier  [30].  Extending  Equation  (3)  for  the  EC, 
UT  and  MW  modalities,  the  decision-level  fusion  images  for  the  multi-modal  case  is 
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mathematically  expressed  by  D{EC,UT} ,  j)^EC  MW^  and  j){ec,ut,mw}  as  sEown  [ n  Equations  (10)- 
(12),  for  the  {EC,UT}  ,  {EC^MW}snd,  {EC^UT^MW}  modality  combinations,  respectively. 


jy{EC ,UT}  _ 


(r-{EC,UT}  x  2  y{EC,UT}  ,  {EC ,UT}  x  2  y{EC,UT} 
\uRx  '  1  Rx Fuzzy  '  *  Fuzzy 


(10) 


jy{EC  ,MW} 


s  {EC , MW}  \2  y{EC,MW}  s  {EC , MW}  x2  y{EC,MW} 

V^Rx  )  1  Rx  Fuzzy  )  1  Fuzzy 

/ -{EC,. MW}  x2  / {EC, MW}  x2 

/  TVufej  / 


(11) 


s  {EC, UT, MW}  x2  y{EC,UT,MW}  s  {EC, UT, MW}  x2  y{EC,UT,MW} 

j^{EC,UT,MW}  _  \aRx _ )  1  Rx _ +  Fuzzy _ '  1  Fuzzy _ 

^  ,  {EC,UT,MW}\2  ./  {EC,UT,MW}\  2  ’ 

'  ^  K.v  J  +  V7 Fuzzy  > 

A  material  loss  estimation  image  was  computed  for  each  of  the  decision-level  fusion  images  for 
the  different  modality  combinations.  The  material  loss  estimation  presented  in  detail  in  [29], 
produces  the  final  material  loss  estimation  images,  p'HC-l,T'<  ?  p{Ec,MW}  an(\pihXU'Evny<  for  the 
various  modality  combinations  {EC,UT}  ,  {EC, MW} and,  {EC,UT,MW}  respectively. 
Examples  of  material  loss  estimation  images  are  shown  in  Figure  13. 


C.  3-D  Model  Generation  and  Fatigue  Testing  of  the  Panels 

ANSYS®  was  to  perform  the  simulated  fatigue  testing  based  on  the  3-D  model  of  the  panel.  The 
simulated  fatigue  testing  process  is  presented  in  the  next  section. 


IV.  Experimental  Results 

A.  Multi-Modal  Data  Fusion  Experiments 

The  NDE  data  collected  from  the  three  modalities  were  subjected  to  the  RX  and  Fuzzy  detectors 
and  then  to  the  decision-level  fusion  process,  the  details  of  which  have  been  explained  in  Section 
III.B.  Experimental  results  for  the  raw  EC,  UT  and  MW  images,  with  their  respective  material 
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loss  estimation  images  are  shown  in  Figures  10-12,  with  the  reference  scale  for  material  loss 
estimation  between  0%  and  20%  are  presented  for  corroded  test  panel  1.  Figure  10  presents  EC 
testing  data  prior  to  round  1  of  actual  fatigue  testing  [11],  Figure  1 1  presents  the  UT  data  and 
Figure  12  shows  the  MW  data. 


O  0  (M 
O  0  Q 
rf)  0  0*1 


(a)  (b) 

Figure  10:  Data  collected  and  data  fusion  based  material  loss  estimation  algorithm  results  for  test 
panel  1  prior  to  round  1  of  actual  fatigue  testing,  (a)  EC  data  for  test  panel  1 ,  (b)  material  loss 

estimation  map  with  reference  scale  [11]. 


i 


(a)  (b) 

Figure  11:  Data  collected  and  data  fusion  based  material  loss  estimation  algorithm  results  for  test 
panel  prior  to  round  1  of  actual  fatigue  testing,  (a)  UT  data  for  test  panel  1,  (b)  material  loss 

estimation  map  with  reference  scale. 


Figure  12:  Data  collected  and  data  fusion  based  material  loss  estimation  algorithm  results  for  test 
panel  1  prior  to  round  1  of  actual  fatigue  testing,  (a)  MW  data  for  test  panel  1,  (b)  material  loss 

estimation  map  with  reference  scale. 
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The  material  loss  estimation  images  using  the  data  fusion  method  for  the  individual  modalities 
were  performed  so  that  the  results  could  be  compared  to  the  multi-modal  data  fusion-based 
material  loss  estimation  method.  Once  the  material  loss  estimation  results  for  all  three  modalities 
were  obtained,  the  multi-modal  data  fusion  algorithm  explained  in  Section  III.B  was  used  to 
obtain  the  material  loss  estimation  images.  For  test  panel  1,  Figure  13(a)  shows  the  EC  and  UT 
fused  results,  Figure  13(b)  shows  the  EC  and  MW  fused  results  while  the  EC,  UT  and  MW  fused 
results  for  the  NDE  data  obtained  prior  to  the  first  round  of  actual  mechanical  testing  are  shown 
in  Figure  13(c). 


■ 


-  12 
-  lO 


(a)  (b)  (c) 

Figure  13:  Multi-modal  data  fusion  based  material  loss  estimation  results  for  panel  1  for  NDE 
data  obtained  prior  to  round  lof  fatigue  testing,  (a)  EC  and  UT  fused  result,  (b)  EC  and  MW 

fused  result,  (c)  EC,  UT  and  MW  fused  result. 


Material  loss  estimation  results  obtained  from  the  above  cases  using  the  individual  modalities 
and  also  the  multi-modal  data  fusion  approach  were  analyzed  with  respect  to  the  actual 
experimental  results.  The  average  material  loss  was  computed  over  the  non- zero  corrosion  areas 
in  the  material  loss  estimation  results  obtained  using  each  of  the  three  NDE  modalities,  EC 
(Figure  1 1(c)),  UT  (Figure  12(c))  and  MW  (Figure  13(c))  and  also  from  the  fused  modalities,  EC 
and  UT  (Figure  14(a)),  EC  and  MW  (Figure  15(b))  and  EC,  UT  and  MW  (Figure  14(c)).  The 
average  material  losses  of  test  panel  1  as  obtained  from  the  various  modalities  and  also  from  the 
fusion  of  the  modalities  are  shown  in  Table  1. 
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Table  1 :  Summary  of  material  loss  estimation  results  using  uni-  and  multi-modal  data  fusion  for 
NDE  data  obtained  prior  to  first  round  of  mechanical  testing 


Modality 

Material  Loss  Estimation  (%) 

EC 

13 

UT 

17.1  (top  scan  only) 

MW 

16.3  (top  scan  only) 

{EC,UT} 

14.2 

{EC,  MW} 

14.7 

{EC,UT,MW} 

17.4 

In  previous  research  test  panel  1  showed  an  actual  average  material  loss  of  16-17%  over  the 
entire  panel  [29],  compared  to  the  material  loss  estimated  based  on  comprehensive  structural 
analysis  process  for  different  modality  combinations  given  in  Table  1.  The  UT  scan  denotes  a 
17.1%  average  material  loss  for  test  panel  1,  which  is  similar  to  the  EC,  UT  and  MW  fused  result 
(17.4%).  It  is  important  to  mention  here  that  the  UT  data  depicts  the  material  loss  present  in  the 
top  layer  of  the  panel,  which  does  not  represent  the  material  loss  of  the  entire  panel  since 
corrosion  exists  in  both  the  top  and  bottom  layer  of  the  panels  (see  Figure  1).  The  MW  images 
also  provide  corrosion/material  loss  information  for  the  top  layer  of  the  test  panels  only.  In  order 
to  evaluate  the  average  material  loss  over  the  entire  panel,  the  EC  images,  which  provide  depth 
information  at  the  various  layers  within  the  panel,  were  fused  with  the  other  modalities  to  obtain 
a  more  comprehensive  material  loss  evaluation. 

B.  Actual  Fatigue  Testing  Results  for  Test  Panels 

During  the  actual  fatigue  testing  of  the  panels  we  observed  that  all  of  the  cracks  developed  on  the 
lower  row  of  rivets,  suggesting  a  higher  stress  concentration  at  this  row.  Table  2  presents  a 
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summary  of  the  actual  fatigue  testing  crack  locations  across  the  two  rounds  of  mechanical 
testing,  the  details  of  which  can  be  found  in  [1 1],  These  results  are  used  for  comparison  purposes 
with  the  crack  locations  predicted  by  the  data  fusion  based  simulated  fatigue  testing  process 
using  FEM  structural  modeling  and  analysis.  For  reference  purposes,  a  schematic  outlining  the 
various  rounds  of  the  NDE  data  collection  and  mechanical  loading  process  is  provided  in  Figure 
2  for  the  different  panels. 

Table  2:  Summary  of  crack  locations  for  the  two  rounds  of  fatigue  loading.  Refer  to  Figure  15 

for  test  panel  location  key. 


Panel  # 

Crack  locations  found  after 

Round  1 

Cracks  locations  found  after 

Round  2 

1 

13,  15 

13,  15 

2 

13 

13,  14 

3 

None 

14 

4 

None 

14 

6 

13,  16 

13,  14, 16 

C.  3-D  Model  Generation  for  Simulated  Fatigue  Testing 

A  3-D  geometrical  model  for  the  riveted  test  panels  was  generated  based  on  a  localized  non¬ 
overlapping  window  method  using  a  material  loss  estimate  map  of  a  given  test  panel.  The 
method  divided  the  height  and  width  dimensions  of  the  test  panel  into  10x10  (approximately  5 
mm  x  5  mm)  non-overlapping  windows.  The  average  material  loss  over  a  non-overlapping 
window  was  estimated,  and  the  3-D  center  of  mass  was  computed.  A  circular  puck-like  region 
was  created  with  a  thickness  equal  to  the  average  material  loss  of  the  window,  and  area  equal  to 
the  number  of  non-zero  pixels  (indicating  corrosion  and  thus  material  loss)  present  in  the 
window.  This  circular  puck-like  region  was  then  centered  within  the  non-overlapping  window. 
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This  method  was  repeated  for  all  non-overlapping  windows  in  each  panel.  Such  a  3-D  model 
obtained  for  test  panel  1  for  the  {EC,UT,MW} modality  combination  is  shown  in  Figure  14, 
illustrated  using  ANSYS®.  Note  that  the  puck-like  regions  shown  in  Figure  14  are  actually  solid 
material  loss  regions,  which  could  not  be  visually  represented  using  the  ANSYS®  display  tool. 
Also,  note  that  the  3-D  modeling  method  was  developed  based  on  constraints  in  ANSYS®,  which 
is  used  for  simulated  fatigue  loading  of  the  test  panels. 


Figure  14:  Model  generation  for  a  test  panel  using  localized  non-overlapping  window  method, 
displayed  using  ANSYS®  for  test  panel  1  for  the  {EC,UT,MW}  modality  combination. 


C.  Simulated  Fatigue  Testing  Results 

As  part  of  the  comprehensive  structural  analysis  process,  the  simulated  fatigue  testing  process 
using  ANSYS  was  performed.  The  riveted  regions  in  the  test  panel  3D  models  were  marked  and 
the  displacements  with  respect  to  each  layer  were  set  to  zero  to  simulate  the  actual  riveted 
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structure  of  the  panels.  After  the  riveted  regions  were  marked  and  their  displacement  set  to  zero 
for  all  directions  at  these  regions,  the  boundary  conditions  applied  to  the  panel  during  simulated 
loading  were  setup  to  simulate  the  actual  mechanical  loading.  In  the  actual  fatigue  testing,  the 
test  panel  was  subjected  to  a  fixed  load  on  the  top  and  an  actuator  at  the  bottom  to  facilitate  the 
cyclical  loading  of  the  panel.  Details  of  the  actual  fatigue  testing  are  provided  in  [1 1], 
Alternating  Stress  Intensity  (Alt-SI)  values  for  all  the  nodes  of  the  meshed  panel  were 
determined  using  the  Fatigue  Analysis  module  in  ANSYS®  as  a  measure  of  the  vulnerability  of  a 
specific  node/region  for  failure.  In  general,  evaluating  the  results  of  the  simulated  structural 
analysis  of  the  test  panels,  it  was  observed  that  some  locations  had  significantly  higher  stress 
concentration  (Alt-SI)  as  compared  to  the  entire  panel.  These  panel  locations  were  designated  as 
the  critical  locations  of  the  panel  (locations  13-18),  as  shown  in  from  Figure  15.  Based  on  this, 
locations  1-12  are  designated  as  the  non-critical  panel  locations. 


Figure  15:  Location  key  (critical  and  non-critical  locations)  for  crack  initiation  sites  in  test 

panels. 
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Table  3:  Summary  of  Alt-SI  values  for  test  panel  1  for  the  two  rounds  of  simulated  fatigue 

testing  using  multi-modal  data  fusion  results. 


Before  round  1  of  actual  testing 

After  round  1  of  actual  testing 

{EC,UT} 

(MPa) 

{EC, MW} 

(MPa) 

{EC,UT,MW} 

(MPa) 

{EC,UT} 

(MPa) 

{EC, MW} 

(MPa) 

{EC,UT,MW} 

(MPa) 

Location  1 

3.2 

2.7 

2.2 

5.5 

4.7 

8.1 

Location  2 

0.2 

1.3 

0.8 

1.0 

0.6 

0.3 

Location  3 

0.7 

1.0 

0.6 

1.1 

0.5 

0.2 

Location  4 

0.5 

1.3 

0.4 

0.7 

0.5 

0.2 

Location  5 

0.2 

1.3 

0.6 

1.0 

0.3 

0.0 

Location  6 

3.9 

2.2 

3.5 

5.5 

9.7 

6.4 

Location  7 

28.7 

28.0 

8.5 

6.8 

2.2 

2.5 

Location  8 

1.1 

0.8 

4.5 

3.7 

3.0 

1.2 

Location  9 

2.1 

6.5 

4.3 

3.8 

5.0 

1.7 

Location  10 

0.4 

1.0 

3.0 

3.6 

4.8 

0.8 

Location  11 

2.7 

1.4 

5.9 

2.8 

1.5 

1.2 

Location  12 

4.8 

5.0 

4.0 

21.0 

5.1 

0.8 

Location  13 

134.9 

109.3 

83.6 

55.6 

84.1 

76.9 

Location  14 

72.9 

63.2 

70.5 

108.1 

67.5 

45.0 

Location  15 

57.5 

75.9 

55.0 

70.7 

64.6 

51.9 

Location  16 

33.2 

92.7 

21.1 

46.5 

54.5 

47.4 

Location  17 

9.2 

63.5 

35.3 

55.4 

47.9 

36.0 

Location  18 

40.0 

34.7 

37.4 

45.4 

54.8 

43.3 

Furthermore,  the  crack  locations  identified  during  the  actual  fatigue  testing  (Table  1)  exhibited 
no  cracks  in  the  test  panels  at  locations  1-12.  All  the  detected  cracks  were  between  locations  13- 
18.  Table  3  gives  the  Alt-SI  values  for  the  different  locations  from  Figure  15  for  test  panel  1  for 
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different  modality  combinations.  Similar  data  tables  were  generated  for  test  panels  1-4  and  the 
pristine  panel.  Table  4  presents  the  Alt-SI  values  for  test  panels  1-4  and  the  pristine  panel  for  the 


{EC,UT}  modality  combination. 

Table  4:  Summary  of  Alt-SI  values  for  test  panels  1-4  and  pristine  panel  for  the  two  rounds  of 
simulated  fatigue  testing  using  multi-modal  data  fusion:  {EC,UT} . 


Before  round  1  of  actual  testing  (in  MPa) 

After  round  1  of  actual  testing  (in  MPa) 

Test 
Panel  1 

Test 
Panel  2 

Test 
Panel  3 

Test 
Panel  4 

Pristine 

Panel 

Test 
Panel  1 

Test 
Panel  2 

Test 
Panel  3 

Test 
Panel  4 

Pristine 

Panel 

Location  1 

3.2 

0.8 

3.4 

1.3 

1.5 

5.5 

2.3 

9.1 

10.0 

12.7 

Location  2 

0.2 

0.5 

4.6 

2.3 

1.0 

1.0 

0.4 

0.7 

0.5 

2.0 

Location  3 

0.7 

0.4 

4.2 

3.3 

0.7 

1.1 

0.4 

0.5 

0.5 

1.7 

Location  4 

0.5 

0.5 

6.0 

1.7 

0.2 

0.7 

0.3 

0.4 

0.7 

1.2 

Location  5 

0.2 

0.6 

5.5 

2.3 

0.5 

1.0 

0.1 

0.1 

0.2 

0.3 

Location  6 

3.9 

2.0 

2.4 

18.8 

1.1 

5.5 

8.0 

1.4 

9.1 

1.6 

Location  7 

28.7 

4.8 

6.8 

33.4 

5.8 

6.8 

11.5 

9.7 

24.3 

3.0 

Location  8 

1.1 

3.3 

3.5 

8.8 

3.5 

3.7 

2.3 

1.6 

3.9 

4.4 

Location  9 

2.1 

2.0 

10.9 

9.2 

1.4 

3.8 

2.4 

2.0 

4.0 

4.3 

Location  10 

0.4 

0.4 

0.9 

7.4 

4.1 

3.6 

11.4 

3.1 

4.9 

4.0 

Location  11 

2.7 

1.6 

2.6 

6.3 

5.5 

2.8 

1.7 

2.2 

3.0 

4.5 

Location  12 

4.8 

1.7 

3.9 

6.4 

5.4 

21.0 

5.1 

3.6 

6.1 

34.1 

Location  13 

134.9 

44.0 

48.7 

56.2 

50.1 

55.6 

73.0 

32.9 

49.4 

92.8 

Location  14 

72.9 

35.6 

33.3 

55.4 

1.0 

108.1 

58.2 

110.9 

56.7 

60.8 

Location  15 

57.5 

39.3 

53.6 

51.4 

34.2 

70.7 

47.0 

56.6 

49.0 

49.1 

Location  16 

33.2 

26.2 

31.3 

55.8 

34.5 

46.5 

43.1 

39.7 

49.3 

67.1 

Location  17 

9.2 

26.1 

36.5 

53.2 

13.2 

55.4 

42.1 

50.1 

53.9 

46.2 

Location  18 

40.0 

53.0 

49.6 

51.1 

24.6 

45.4 

56.3 

53.2 

61.7 

59.5 

A  fuzzy-set  approach  was  used  to  find  the  cutoff  value  for  automatic  detection  of  the  cracks  from 
the  Alt-SI  values  of  the  simulated  fatigue  testing  of  the  panels.  A  fuzzy  set  P  provides  the 
degree  of  vulnerability  in  a  structure  (test  and  pristine  panels)  at  a  specified  position  in  terms  of 
Alt-SI.  Let  jup(V )  represents  the  membership  value  in  P  for  an  Alt-SI  value,  V ,  determined  in 

a  structure  at  a  specified  position.  jup(V)  is  defined  in  equation  (13): 
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M^)  = 


— ,  if  Alt  -SI <70 MPa 
70 

1,  otherwise 


(13) 


Figure  16:  Fuzzy-logic  based  method  of  automatically  finding  the  threshold  for  crack  prediction. 


Figure  16  shows  the  membership  function jup(V)  .  Alpha-cuts  (a )  on  the  fuzzy  set  P  were 
examined  for  determining  test  panel  vulnerability  positions,  where  all  positions  with  //P(K)  >  a 
are  designated  as  vulnerable  forO  <  a  <  1  as  shown  in  Equation  14. 

fl,  if  //p(V)  >  a\ 

Vulnerability  =  ■<  >for0<a<l  (14) 

[0,  otherwise  J 

An  alpha-cut  value  of  0.45  (a  =  0.45 )  produced  the  best  results,  the  details  of  which  for  each  test 
panel  and  each  modality  combination  for  all  the  two  rounds  of  simulated  fatigue  testing  are 
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summarized  in  Table  5.  True  positive  and  true  negative  crack  location  detection  rates  were 
computed  over  the  four  corroded  and  one  pristine  test  panels.  A  true  positive  crack  location  is 
correctly  labeled  if  the  location  satisfies  the  Alt-SI  threshold  constraint,  and  the  location  in  the 
test  panel  for  the  corresponding  round  of  mechanical  loading  is  designated  as  a  crack  site,  as 
given  in  Table  2.  A  true  negative  location  is  labeled  correctly  if  the  location  does  not  satisfy  the 
Alt-SI  threshold  constraint,  and  the  location  in  the  test  panel  for  the  corresponding  round  of 
mechanical  loading  is  not  called  a  crack  site,  as  shown  in  Table  2.  False  positives  are  the  number 
of  cracks  that  were  erroneously  detected  across  all  panels,  and  false  negatives  refer  to  cases  when 
the  simulated  fatigue  loading  method  failed  to  predict  a  crack  location  that  was  actually  a  crack 
location  obtained  by  the  mechanical  loading  process.  For  each  round  of  testing,  and  for  each 
modality  combination,  18  total  number  of  true  positive  and  true  negative  locations  resulted.  In 
Table  5,  the  number  on  the  left  side  of  the  slash  indicates  the  number  of  vulnerable  locations 
(true  positive)  and  the  number  of  non- vulnerable  locations  (true  negative)  found  from  the 
simulated  fatigue  testing  process,  and  the  number  to  the  right  of  the  slash  gives  the  true  positive 
and  true  negative  locations  as  found  from  actual  fatigue  testing. 


Table  5:  Accuracy  metrics  of  simulated  testing  v/s  mechanical  testing.  Number  of  locations 
found  using  simulated  fatigue  testing/number  of  locations  using  actual  fatigue  testing. 


Before  round  1  of  actual  testing 

After  round  1  of  actual  testing 

True  Positive 

True  Negative 

True  Positive 

True  Negative 

{EC,UT} 

{EC, MW} 

{EC,UT,MW} 

{EC,UT} 

{EC, MW} 

{EC,UT,MW} 

{EC,UT} 

{EC, MW} 

{EC,UT,MW} 

{EC,UT} 

{EC, MW} 

{EC,UT,MW} 

Panel  1 

2/2 

2/2 

2/2 

14/16 

12/16 

13/16 

2/2 

2/2 

2/2 

12/16 

12/16 

12/16 

Panel  2 

i/i 

i/i 

i/i 

17/17 

17/17 

17/17 

2/2 

2/2 

2/2 

16/16 

16/16 

16/16 

Panel  3 

0/0 

0/0 

0/0 

18/18 

18/18 

18/18 

1/1 

1/1 

1/1 

17/17 

17/17 

17/17 

Panel  4 

0/0 

0/0 

0/0 

18/18 

18/18 

18/18 

1/1 

1/1 

1/1 

17/17 

17/17 

17/17 

pristine 

2/2 

2/2 

1/2 

16/16 

16/16 

16/16 

3/3 

2/3 

1/3 

15/15 

15/15 

15/15 

Total 

5/5 

5/5 

4/5 

83/85 

81/85 

82/85 

9/9 

8/9 

7/9 

77/81 

77/81 

77/81 
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The  accuracy  for  each  round  of  simulated  fatigue  testing  is  computed  by  calculating  the  sum  of 
true  positives  and  true  negatives  detected  divided  by  the  total  number  of  positions  for  each  round 
of  the  fatigue  testing.  Table  6  shows  the  summarized  results  obtained  using  this  method.  From 
[1 1],  the  EC  only  data  the  correct  prediction  rates  were  as  high  as  88.9%  for  the  rounds  of 
simulated  fatiguing,  with  an  average  correct  prediction  rate  of  87.8%.  For  the  multi-modal  fused 
data,  the  correct  prediction  rate  was  as  high  as  98.8%  for  the  {EC,UT}  case  before  round  1  of 
actual  mechanical  testing.  For  each  round  of  simulated  fatigue  testing,  the  Average  Accuracy 
was  computed  by  averaging  the  Overall  Accuracies  for  the  different  modality  combinations.  As 
shown  in  Table  6,  the  average  correct  prediction  rates  before  the  first  round  of  actual  testing  and 
after  the  first  round  of  actual  testing  are  95.0%  and  92.0%,  respectively.  In  general,  the  data 
fusion  process  yielded  a  consistent  correlation  for  vulnerability  location  compared  to  visual 
inspection  from  actual  fatigue  testing. 


Table  6:  Summary  of  accuracy  estimates  obtained  using  information  from  Table  5. 


{EC,UT} 

{EC, MW] 

{EC,UT,MW} 

True 

Positive 

True 

Negative 

Overall 

Accuracy 

True 

Positive 

True 

Negative 

Overall 

Accuracy 

True 

Positive 

True 

Negative 

Overall 

Accuracy 

Before 
round  1 
of  actual 
testing 

100.0% 

97.6% 

98.8% 

100.0% 

95.3% 

97.7% 

80.0% 

96.5% 

88.2% 

Average 

Accuracy 

(98. 8+97. 7+88. 2)/3=95.0% 

After 
round  1 
of  actual 
testing 

100.0% 

95.1% 

97.5% 

88.9% 

95.1% 

92.0% 

77.8% 

95.1% 

86.5% 

Average 

Accuracy 

(97. 5+92+86. 5)/3=92.0% 
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V.  Discussion 


From  Figure  13  for  test  panel  1,  the  modality  combination  {EC,UT,MW}  visually  detects 
greater  corrosion  (i.e.,  more  material  loss),  compared  to  the  {EC,UT}  and  {EC, MW} 
combinations.  The  results  in  Table  5  for  test  panel  1  show  that  the  same  vulnerable  locations 
(see  Figure  15)  were  found  for  the  {EC,UT},  {EC, MW} ,  and  {EC,UT,MW}  combinations 
based  on  the  comprehensive  structural  analysis  process.  However,  from  Table  6,  the  overall 
vulnerable  location  estimate  results  were  slightly  lower  for  the  {EC,UT,MW}  combination 
than  for  {EC,UT}.  From  Table  5,  the  {EC,UT}  and  {EC,UT,MW}  combinations  correctly 
found  the  same  vulnerable  locations  in  the  corroded  test  panels,  but  the  {EC,UT}  combination 
yielded  better  results  than  the  {EC,UT,MW}  combination  for  finding  vulnerable  locations  in 
the  pristine  panel  for  both  rounds  of  testing.  From  [11],  the  pristine  panel  was  slightly  bent 
during  the  first  round  of  mechanical  loading,  resulting  in  two  crack  locations,  and  a  third  crack 
was  obtained  during  the  second  round  of  mechanical  loading  (see  Table  2).  Because  there  was 
no  surface  corrosion  present  at  those  crack  locations,  the  MW  modality  was  not  able  to 
characterize  the  cracks  and  may  have  skewed  the  fused  model  for  pristine  panel  for  the 
{EC,UT,MW}  combination.  In  general,  fatigue  is  highly  sensitive  to  small  surface  features 
and  discontinuities  and  is  therefore  statistically  varied.  To  compound  the  variability,  the 
complexity  of  the  riveted  panels  used  in  this  investigation  incorporates  many  variables  that  may 
have  affected  fatigue  of  the  panels  such  as  residual  stresses,  defects,  and  misalignment  possibly 
incorporated  during  panel  assembly.  Naturally,  one  would  expect  that  the  multi-modal  fused 
model  using  the  most  number  of  modalities  would  produce  the  best  results  because  it  utilizes 
complementary  information  from  the  various  modalities  to  create  a  more  accurate  picture  of  the 
corrosion  environment  and  thus  provide  the  best  understanding  of  the  fatigue  behavior  as  well.  In 
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our  case,  the  combination  of  two  modalities  ({EC,UT}  or  {EC,  MW})  produced  the  best  results 
as  shown  in  Table  6  as  compared  to  using  all  the  three  modalities.  This  might  be  attributed  to  the 
fact  that  the  FEM  based  model  created  from  nondestructive  testing  data  is  limited  in  its  ability  to 
predict  fatigue.  The  inconsistency  of  modeling  predictions  is  likely  attributable  to  the 
dependency  of  model  stresses  on  the  location  from  which  data  is  obtained.  A  predictive  fatigue 
model  would  require  consistently  refined  meshing  that  would  allow  for  stress  data  to  be  obtained 
from  the  same  location  between  modeling  increments.  This  was  not  possible  in  our  case  as  with 
each  round  of  NDE  data  collection  new  3-D  models  were  generated  using  the  testing  data 
obtained  from  that  particular  round  to  be  used  for  the  FEM  based  fatigue  analysis.  Given  that 
fatigue  cracks  generally  nucleate  in  isolated  locations  near  stress  concentration  sites  (e.g.  rivet 
holes),  a  model  with  refined  meshing  only  in  the  fatigue  sensitive  regions  may  improve  the 
model  accuracy  without  significant  increase  to  the  computing  power  required. 

Future  research  will  explore  different  types  of  data  fusion  approaches  to  combine  NDE  modality 
data  to  characterize  the  test  structures  to  attempt  to  integrate  unique  information  from  each 
modality.  In  order  to  address  the  limitations  of  FEM-based  methods  in  their  ability  to  predict 
fatigue,  more  specialized  numerical  modeling  techniques  need  to  be  investigated  which  would 
allow  the  generation  of  a  more  consistently  structured  mesh  or  a  consistently  refined  mesh  in 
which  it  is  possible  to  acquire  stress  data  from  the  same  location  between  meshing  increments, 
which  would  allow  for  more  consistent  prediction  of  stresses. 
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